System and method of determining a risk score for triage

ABSTRACT

The present disclosure provides a system and method of determining a risk score for triage. In particular, a system is provided for providing an assessment of risk of a cardiac event for a patient, for example an incoming patient to a hospital emergency department complaining of chest pain. In the disclosure, the system includes an input device for measuring physiological data based vital signs parameter of the patient, a twelve-lead electrocardiogram (ECG) device for establishing an ECG obtained from results of the electrocardiography procedure, and determining an ECG parameter and a heart rate variability (HRV) parameter therefrom. An ensemble-based scoring system is further provided, establishing weighted classifier based on past patient data and where the vital signs parameter, the ECG parameter and the HRV parameter are compared to corresponding weighted classifiers to determine a risk score. A corresponding method to determine a risk score for triage is also provided.

TECHNICAL FIELD

The present disclosure relates generally to a system and a method ofdetermining a risk score for triage. More particularly, embodiments ofthe present disclosure are directed to systems and techniques ofdetermining a cardiac risk score, such as for a hospital emergencydepartment, ambulance, clinic, ward, intensive care unit (ICU) or home,based on measurements and readings taken.

BACKGROUND

Patients seen at the emergency department (ED) of a hospital complainingof chest pain have varying levels of complication risk in the acutephase of treatment, usually identified as the first 72 hours from whenthe affliction first occurs. Similarly, cardiac risk screening andmonitoring is often done in ambulances, clinics, wards, ICUs and evenfrom home. In the ED, triage is carried out to assess the severity ofthe incoming patient's condition and to assign appropriate treatmentpriorities. Early stratification of risk improves treatment strategiesas well as assists with the formulation of proper monitoring for thepatient. Risk stratification is necessary in EDs as medical resources,such as doctors, nurses, monitoring systems, monitored beds,resuscitation facilities, intensive care units, etc., are neversufficient for all incoming patients to be attended to instantaneously.For example, in the United States, approximately 6 million patientspresent with chest pain to the ED each year, which makes chest pain oneof the leading principle diagnoses during ED visits. Similarly, earlyidentification of high risk patients can benefit management inambulances, clinics, wards, ICUs and even for home monitoring. Chestpain severity ranges from self-limited to severe and life threateningsituations such as cardiac arrest and lethal arrhythmias. The need toidentify high-risk patients allows for timely intervention forpreventable and treatable complications.

In the past few decades, scoring systems have also been developed, andare now widely used in intensive care units (ICUs) to predict clinicaloutcomes and assess the severity of illnesses.

Some of the systems which have been developed are for example, AcutePhysiology and Chronic Health Evaluation (APACHE), Simplified AcutePhysiology Score (SAPS) and Mortality Probability Model (MPM). Eachscoring system has a specific purpose and its own range of applications.For example, risk of death, organ dysfunction assessment and severity ofillnesses are possible outcomes of some of these scoring systems.

The development of scoring systems relies on the appropriate selectionof variables or parameters with which prediction outcomes areassociated. Present triage tools and risk-stratification systems forpatients with suspected acute coronary syndromes (ACS) are based on acombination of traditional clinical factors such as patient medicalhistory, cardiac bio markers, and measurements obtained from ED incomingpatient screenings, for example observing and obtaining traditionalvital signs such as heart rate, respiratory rate, blood pressure,temperature, and pulse oximetry. However, these parameters have not beenshown to correlate well with short or long-term clinical outcomes.

Presently, although thrombolysis in myocardial infarction (TIMI) riskscore is currently the most clinically accepted risk categorization ofpatients with ACS, its prediction accuracy is debatable and perhapssomewhat controversial. There are as such limitations to current riskscores for prediction of cardiovascular complications, whilst at thesame time, clinical judgment is subjective, as well as being hampered bya limitation in doctoral resource.

SUMMARY

A system is provided for determining a risk score (e.g. for triage),including: a first input device for measuring a first input parameterrelating to physiological data of a patient, the first input parameterincluding a vital signs parameter; a twelve-lead electrodeelectrocardiogram (ECG) device, for carrying out a electrocardiographyprocedure on the patient, and establishing an ECG obtained from resultsof the electrocardiography procedure, the ECG device including an ECGextraction module to extract at least one ECG parameter from the ECG; aheart rate variability (HRV) analysis module for determining a HRVanalysis from the ECG, the HRV analysis including at least one HRVparameter; and an ensemble-based scoring system, including: a pluralityof weighted classifiers for providing a risk score calculation, theplurality of weighted classifiers established based on past patient datain a database of accumulated past patient data; and an analysis modulefor receiving the first input parameter, the at least one HRV parameter,and the at least one ECG parameter which are communicated or transmittedto the ensemble-based scoring system, wherein the analysis moduledetermines a risk score by comparing the first input parameter, the atleast one HRV parameter, and the at least one ECG parameter tocorresponding weighted classifiers.

BRIEF DESCRIPTION OF DRAWINGS

Embodiments of the present disclosure are explained, by way of example,and with reference to the accompanying drawings. It is to be noted thatthe appended drawings illustrate only examples of embodiments of thisdisclosure and are therefore not to be considered limiting of its scope,for the disclosure may admit to other equally effective embodiments.

FIG. 1 illustrates a patient connected to a 12-lead ECG.

FIG. 2 illustrates a system architecture of a triage system of thepresent embodiment.

FIG. 3A illustrates a sample ECG printout as provided by the triagesystem of the present embodiment.

FIG. 3B illustrates a close up of a cardiac cycle from the ECG of FIG.3A.

FIG. 3C illustrates a close up of the QRS complex of the cardiac cycleof FIG. 3B.

FIG. 4A illustrates a general structure of ensemble learning basedsystem.

FIG. 4B illustrates a first scoring system utilizing an under-samplingmethod to calculate a risk score.

FIG. 4C illustrates a second scoring system utilizing a hybrid-samplingapproach according to a second embodiment.

FIG. 5A is a block diagram of an algorithm of the under-sampling basedscoring system.

FIG. 5B is a block diagram of an algorithm of the hybrid-sampling basedscoring system.

FIG. 6 illustrates a modular layout of an ensemble-based scoring systemof the triage system according to an embodiment.

FIG. 7 illustrates a modular layout of an ensemble-based scoring systemof the triage system according to a second embodiment.

FIG. 8A charts the performance of USS vs. TIMI and MEWS.

FIG. 8B charts the performance of HSS vs. TIMI and MEWS.

FIG. 9A charts the performance of USS with and without 12-lead ECGparameters.

FIG. 9B charts the performance of HSS with and without 12-lead ECGparameters.

DETAILED DESCRIPTION

In the following, reference is made to embodiments of the disclosure.However, it should be understood that the disclosure is not limited tospecific described embodiments. Instead, any combination of thefollowing features and elements, whether related to differentembodiments or not, is contemplated to implement and practice thedisclosure.

Furthermore, in various embodiments the disclosure provides numerousadvantages over the prior art. However, although embodiments of thedisclosure may achieve advantages over other possible solutions and/orover the prior art, whether or not a particular advantage is achieved bya given embodiment is not limiting of the disclosure. Thus, thefollowing aspects, features, embodiments and advantages are merelyillustrative and are not considered elements or limitations of theappended claims except where explicitly recited in a claim(s). Likewise,any reference to “the invention” shall not be construed as ageneralization of any inventive subject matter disclosed herein andshall not be considered to be an element or limitation of the appendedclaims except where explicitly recited in a claim(s).

In an aspect of the present disclosure, there is provided a system fordetermining a risk score for triage or for otherenvironments/situations, the system including a first input device formeasuring a first input parameter relating to physiological data of apatient, the first input parameter including a vital signs parameter; atwelve-lead electrode electrocardiogram (ECG) device, for carrying out aelectrocardiography procedure on the patient, and establishing an ECGobtained from results of the electrocardiography procedure, the ECGdevice including an ECG extraction module to extract at least one ECGparameter from the ECG; a heart rate variability (HRV) analysis modulefor determining a HRV analysis from the ECG, the HRV analysis includingat least one HRV parameter; and an ensemble-based scoring system,including: a plurality of weighted classifiers for providing a riskscore calculation, the plurality of weighted classifiers establishedbased on past patient data in a database of accumulated past patientdata; and an analysis module for receiving the first input parameter,the at least one HRV parameter, and the at least one ECG parameter whichare communicated or transmitted to the ensemble-based scoring system,wherein the analysis module determines a risk score by comparing thefirst input parameter, the at least one HRV parameter, and the at leastone ECG parameter to corresponding weighted classifiers.

Such a system can provide a prompt insight suitable for use in ahospital emergency department, ambulance, clinic, ward, intensive careunit (ICU) or home in providing a classification as to the severity of acardiac event situation for an incoming patient. The present systemcaters to the utilization of a combination of a vital signs parameter, aECG parameter and a HRV parameter, which has been proposed after muchstudy and research by the present inventors to provide a possiblyclearer insight as to the identification of risk of an acute coronarysyndrome. A 12-lead ECG is also proposed for usage in preference to thehigh level of detail and insight such a procedure provides.

Furthermore the present system provides for an ensemble-based scoringsystem, which is inherently an intelligent artificial neural network,capable of learning and training from past data to establish classifiersthat are weighted based on a derived understanding of which parameterscontribute to a higher occurrence or severity of cardiac events, and forcomparison with present input parameters to determine a risk score as totriage.

According to an embodiment, the at least one ECG parameter is any one ofa ST elevation, a T wave inversion, a Q wave, a QT interval correction(QTc), a QRS axis, a left bundle branch block (BBB), a right BBB, anIntraVentricular Conduction Delay (IVCD), a left atrial abnormality(LAA), a left ventricular hypertrophy (VH), a right VH, and an atrialfibrillation.

According to an embodiment, the HRV analysis module further includes atime domain analysis module for determining the at least one HRVparameter from a plurality of RR intervals extracted from the ECG.

According to an embodiment, the at least one HRV parameter is any one ofan average length of the RR intervals, standard deviation of all RRintervals, a mean heart rate, a standard deviation of all instantaneousheart rate values, a NN50 count, a pNN50 percentage, a square root ofmean squared differences of successive RR intervals, a HRV triangularindex, and a baseline width of triangular fit into a RR intervalhistogram.

According to an embodiment, the HRV analysis module further includes afrequency domain analysis module for determining the at least one HRVparameter from a plurality of RR intervals extracted from the ECG.

According to an embodiment, the at least one HRV parameter is any one ofa total power, a very low frequency power, a low frequency power (LF), ahigh frequency power (HF), a normalized low frequency power, anormalized high frequency power, and a ratio of LF/HF.

According to an embodiment, the first input device is any one of a heartrate monitor, a respiratory rate monitor, a blood pressure monitor, anoximeter, and a dolorimeter.

According to an embodiment, there is provided a second input device forestablishing a second input parameter relating to a medical status ofthe patient, the second input parameter received by the analysis module,and wherein the analysis module determines the risk score by furthercomparing the second input parameter to a corresponding weightedclassifier.

According to an embodiment, the ensemble-based scoring system furtherincludes a data access module for obtaining past patient data andconfigured for data communication with the database of accumulated pastpatient data.

According to an embodiment, the ensemble-based scoring system furtherincludes a sorting module arranged to receive data from the database ofaccumulated past patient data; and sort the data into a plurality ofdata sets, each data set corresponding to a classifier, and including animbalanced data set.

According to an embodiment, the ensemble-based scoring system furtherincludes a sampling module arranged to receive a first imbalanced dataset corresponding to a first classifier, including a first majority dataset including a first number of data samples, and a first minority dataset including a second number of data samples, from the sorting module;and extract a first majority data subset including a third number ofsamples from the first majority data set; wherein the third number ofsamples in the first majority data subset is equal to the second numberof samples in the first minority data set.

According to an embodiment, the ensemble-based scoring system furtherincludes a classifier generation module for establishing the pluralityof weighted classifiers, based on past patient data provided by the dataaccess module.

According to an embodiment, the classifier generation module furtherincludes a training module arranged to receive the first majority datasubset and the first minority data set from the sampling module; andbuild a first classification model to represent the first classifierwith the first majority data subset and the first minority data set.

According to an embodiment, the training module receives a plurality ofmajority data subsets and minority data sets from the sampling module;and builds classification models representing a plurality of classifierswith the received plurality of majority data subsets and minority datasets.

According to an embodiment, the classifier generation module furtherincludes a weighing module for allocating each of the plurality ofclassifiers with an equal weightage, to obtain the plurality of weightedclassifiers.

According to an embodiment, the training module includes a supportvector machine to build the classification model.

According to an embodiment, the ensemble-based system further includesan over-sampling module arranged to receive the first majority datasubset and the first minority data set from the sampling module; andcreate a first synthetic data set by applying a process of syntheticover-sampling with replacement on the first majority data subset and thefirst minority data set.

According to an embodiment, the over-sampling module over-samples thefirst minority data set by taking a data point in the first minoritydata set and introduces synthetic examples along a line segment joiningthe data point to a predetermined number of data point neighbors.

According to an embodiment, the ensemble-based system further includes avalidation module arranged to: build a first classification model with atraining module based on the first majority data subset and the firstminority data set corresponding to the first classifier; validate thefirst classification model against the first synthetic data set; andobtain a resultant prediction accuracy of the first classificationmodel, representing the importance of the first classifier.

According to an embodiment, the over-sampling module receives aplurality of majority data subsets and minority data sets from thesampling module and creates a plurality of synthetic data sets; thetraining module builds a plurality of classification models representinga plurality of classifiers with the received plurality of majority datasubsets and minority data sets; the validation module validates theplurality of classification models against the plurality of syntheticdata sets, and obtains a plurality of prediction accuracies of theclassification models, representing the importance of each of theplurality of classifiers.

According to an embodiment, the classifier generation module furtherincludes a weighing module for allocating each of the plurality ofclassifiers with a weightage according to its importance, to obtain theplurality of weighted classifiers.

According to an embodiment, the analysis module further includes atesting module arranged to: receive any of the first input parameter,the at least one HRV parameter, and the at least one ECG parameter;evaluating the parameter with its corresponding weighted classifier; andgenerate a binary prediction output of either 0 or 1 for each evaluatedweighted classifier.

According to an embodiment, the analysis module further includes ascoring module for calculating the risk score based on a normalizedsummation of the binary prediction outputs of all evaluated weightedclassifiers.

According to a second aspect of the present disclosure, there isprovided a method of determining a risk score for triage, including:measuring a first input parameter relating to physiological data of apatient, the first input parameter including a vital signs parameter;carrying out a twelve-lead electrocardiography procedure on the patient;establishing an ECG (providing ECG information, signals or data) fromresults of the electrocardiography procedure, the ECG including at leastone ECG parameter extractable from the ECG; extracting the at least oneECG parameter from the ECG; determining a heart rate variability (HRV)analysis from the ECG, the HRV analysis including at least one HRVparameter; providing the first input parameter, the second inputparameter, the at least one HRV parameter, and the at least one ECGparameter to an ensemble-based scoring system; the ensemble-basedscoring system including a plurality of weighted classifiers forproviding a risk score calculation, the plurality of weightedclassifiers established based on past patient data in a database ofaccumulated past patient data; and determining a risk score with theensemble-based scoring system by comparing the first input parameter,the at least one HRV parameter, and the at least one ECG parameter tocorresponding weighted classifiers.

According to an embodiment, the at least one ECG parameter is any one ofa ST elevation, a T wave inversion, a Q wave, a QT interval correction(QTc), a QRS axis, a left bundle branch block (BBB), a right BBB, anIntraVentricular Conduction Delay (IVCD), a left atrial abnormality(LAA), a left ventricular hypertrophy (VH), a right VH, and an atrialfibrillation.

According to an embodiment, the method includes extracting a pluralityof RR intervals from the ECG and performing any one of a time domainanalysis and a frequency domain analysis to obtain the at least one HRVparameter.

According to an embodiment, the at least one HRV parameter is any one ofan average length of the RR intervals, standard deviation of all RRintervals, a mean heart rate, a standard deviation of all instantaneousheart rate values, a NN50 count, a pNN50 percentage, a square root ofmean squared differences of successive RR intervals, a HRV triangularindex, a baseline width of triangular fit into a RR interval histogram,a total power, a very low frequency power, a low frequency power (LF), ahigh frequency power (HF), a normalized low frequency power, anormalized high frequency power, and a ratio of LF/HF.

According to an embodiment, the first input parameter is any one of aheart rate, a respiratory rate, a blood pressure reading, a temperaturereading, a Glasgow Coma Score (GCS), an oxygen saturation reading, and apain score.

According to an embodiment, the method includes establishing a secondinput parameter relating to a medical status of the patient, providingthe second input parameter to the ensemble-based scoring system, anddetermining the risk score by further comparing the second inputparameter to a corresponding weighted classifier.

According to an embodiment, the second input parameter is any one of amedical history, a drug history, a smoking history, a family history ofheart disease, and a number of angina events in the past 24 hours.

According to an embodiment, the method includes obtaining the third setof input parameters from a twelve-lead electrocardiography procedure ofat least 5 minutes.

According to an embodiment, the method includes obtaining past patientdata with a data access module, the data access module configured fordata communication with the database of accumulated past patient data.

According to an embodiment, the method includes: receiving data from thedatabase of accumulated past patient data; and sorting the data into aplurality of data sets, each data set corresponding to a classifier, andincluding an imbalanced data set.

According to an embodiment, the method includes receiving a firstimbalanced data set corresponding to a first classifier, including afirst majority data set including a first number of data samples, and afirst minority data set including a second number of data samples, fromthe sorting module; and extracting a first majority data subsetincluding a third number of samples from the first majority data set;wherein the third number of samples in the first majority data subset isequal to the second number of samples in the first minority data set.

According to an embodiment, the method includes establishing theplurality of weighted classifiers with a classifier generation moduleand based on past patient data provided by the data access module.

According to an embodiment, receiving the first majority data subset andthe first minority data set from the sampling module with a trainingmodule; and building a first classification model to represent the firstclassifier with the first majority data subset and the first minoritydata set.

According to an embodiment, the method includes: receiving, with thetraining module, a plurality of majority data subsets and minority datasets from the sampling module; and building classification modelsrepresenting a plurality of classifiers with the received plurality ofmajority data subsets and minority data sets.

According to an embodiment, the method includes allocating, with aweighing module, each of the plurality of classifiers with an equalweightage, to obtain the plurality of weighted classifiers.

According to an embodiment, the method includes building theclassification model with a support vector machine.

According to an embodiment, the method includes: receiving, with anover-sampling module, the first majority data subset and the firstminority data set from the sampling module; and creating a firstsynthetic data set by applying a process of synthetic over-sampling withreplacement on the first majority data subset and the first minoritydata set.

According to an embodiment, the method includes over-sampling the firstminority data set by taking a data point in the first minority data setand introducing synthetic examples along a line segment joining the datapoint to a predetermined number of data point neighbors.

According to an embodiment, the method includes: building a firstclassification model with a training module based on the first majoritydata subset and the first minority data set corresponding to the firstclassifier; validating the first classification model against the firstsynthetic data set with a validation module; and obtaining a resultantprediction accuracy of the first classification model, representing theimportance of the first classifier.

According to an embodiment, the method includes: receiving, with theover-sampling module, a plurality of majority data subsets and minoritydata sets from the sampling module; and creating, with the over-samplingmodule, a plurality of synthetic data sets; building, with the trainingmodule, a plurality of classification models representing a plurality ofclassifiers with the received plurality of majority data subsets andminority data sets; validating, with the validation module, theplurality of classification models against the plurality of syntheticdata sets, and obtains a plurality of prediction accuracies of theclassification models, representing the importance of each of theplurality of classifiers.

According to an embodiment, the method includes allocating, with aweighing module, each of the plurality of classifiers with a weightageaccording to its importance, to obtain the plurality of weightedclassifiers.

According to an embodiment, the method includes: receiving, with atesting module, any of the first input parameter, the at least one HRVparameter, and the at least one ECG parameter; evaluating the parameterwith its corresponding weighted classifier; and generating a binaryprediction output of either 0 or 1 for each evaluated weightedclassifier.

According to an embodiment, the method includes calculating, with ascoring module, the risk score based on a normalized summation of thebinary prediction outputs of all evaluated weighted classifiers.

According to a third aspect of the present disclosure, there is provideda method of determining a risk score, including: receiving a firstimbalanced dataset corresponding to a first classifier and sampling thedata samples to form a first balanced data set; creating a firstsynthetic data set by applying a process of synthetic over-sampling withreplacement on the first balanced data set; building a firstclassification model based on the first balanced data set correspondingto the first classifier; validating the first classification modelagainst the first synthetic data set; and obtaining a resultantprediction accuracy of the first classification model, representing theimportance of the first classifier.

According to an embodiment, the method includes: obtaining a pluralityof prediction accuracies relating to a plurality of classificationmodels; sorting the plurality of classification models according to itsprediction accuracy; allocating each of the plurality of classificationmodels with a weightage according to its importance to obtain aplurality of weighted classifiers; evaluating an input parameter withits corresponding weighted classifier; and generating a binaryprediction output of either 0 or 1 for each evaluated weightedclassifier; and calculating the risk score based on a normalizedsummation of the binary prediction outputs of all evaluated weightedclassifiers.

According to a fourth aspect of the present disclosure, there isprovided a system for determining a cardiac event risk score, including:an ensemble-based scoring system, configured to receive any one of (a) avital signs parameter, (b) an ECG parameter extracted from an ECGestablished by carrying out an ECG procedure, and (c) a HRV parameterdetermined from a HRV analysis of the ECG, the ensemble-based scoringsystem including: a plurality of weighted classifiers for providing arisk score calculation, the plurality of weighted classifiersestablished based on past patient data in a database of accumulated pastpatient data; and an analysis module for receiving the at least one HRVparameter, and the at least one ECG parameter transmitted to theensemble-based scoring system, wherein the analysis module determines acardiac event risk score by comparing any one of the vital signsparameter, the at least one HRV parameter, and the at least one ECGparameter, to corresponding weighted classifiers.

In the present disclosure, depiction of a given element or considerationor use of a particular element number in a particular FIG. or areference thereto in corresponding descriptive material can encompassthe same, an equivalent, or an analogous element or element numberidentified in another FIG. or descriptive material associated therewith.The use of “/” herein means “and/or” unless specifically indicatedotherwise.

As used herein, the term “set” corresponds to or is defined as anon-empty finite organization of elements that mathematically exhibits acardinality of at least 1 (i.e., a set as defined herein can correspondto a singlet or single element set, or a multiple element set), inaccordance with known mathematical definitions (for instance, in amanner corresponding to that described in An Introduction toMathematical Reasoning: Numbers, Sets, and Functions, “Chapter 11:Properties of Finite Sets” (e.g., as indicated on p. 140), by Peter J.Eccles, Cambridge University Press (1998)). In general, an element of aset can include or be a system, an apparatus, a device, a structure, astructural feature, an object, a process, a physical parameter, or avalue depending upon the type of set under consideration.

The terms “group” and “gang” as used herein correspond to or are definedas an organization of two or more elements, e.g., a group or gang can bedefined as a set having at least two components. The term “subgroup” asused herein corresponds to or is defined as a portion of a group organg, and hence corresponds to or can be defined as an organization ofat least one element, e.g., a subgroup can be defined as a set having atleast one component.

Triage Systems

In a present embodiment of the disclosure, there is provided a systemfor determining a risk score for a hospital emergency department triage,in particular triage for acute coronary syndromes (ACS) which includeany group of symptoms attributed to the obstruction of the coronaryarteries. An embodiment of the disclosure can also determine a riskscore in an environment other than a hospital emergency departmenttriage, for instance, in an ambulance, clinic, ward, intensive care unit(ICU), and home or non-medical workplace environment.

Patients presenting to the ED of a hospital with chest pain have varyinglevels of risk of complications in the acute phase of treatment (<72hours). With regard to a hospital ED having limited resources,especially doctors and specialists, it is desired, and in fact necessaryfor the incoming patient to undergo risk stratification with a triageprocess to efficiently address the patient's situation as well as forefficient allocation of resources.

Present scoring systems, as mentioned, do not effectively provideinsight and risk assessment, especially in the area of ACS. Thethrombolysis in myocardial infarction (TIMI) risk assessment iscurrently the most clinically accepted risk categorization of patientscomplaining of ACS. However, the prediction accuracy is being put toquestion by part of the scientific community, for example, in Hess, etal's paper on “Prospective validation of a modified thrombolysis inmyocardial infarction risk score in emergency department patients withchest pain and possible acute coronary syndrome”. In this paper, Hess etal. identified that “a modified TIMI score assessment outperformed theoriginal with regard to diagnostic accuracy . . . . However, both scoresare insufficiently sensitive and specific to recommend as the sole meansof determining disposition in ED chest pain patients”.

Electrocardiography is a non-invasive procedure used by medicalprofessionals to obtain a measure of the electrical activity of apatient's heart, and is carried out by attaching electrodes to the outersurface of a patient's body, and processed by an external processingsystem. The resultant interpretation of the electrical signals is termedas an electrocardiogram (ECG) and is able to provide insight into anyabnormal functionality or rhythms of the patient's heart.

Detection of the heart's electrical activity is provided by attachedelectrodes. In general, the more electrodes are attached, the moreinformation that may be obtained from the resultant readings. Particularconfigurations of electrodes have been established in terms of “leads”,where a lead refers to the tracing of voltage difference between twoelectrodes. Typically, configurations are provided for 3, 5, or 12 leadECGs, where 3 electrodes are used to obtain a 3-lead ECG, 5 electrodesare used to obtain a 5-lead ECG, and 10 electrodes are used to obtain a12-lead ECG.

FIG. 1 illustrates a patient 10 connected to the ten electrodes 12needed for a 12-lead ECG 14. Multiple electrodes are required at variouslocations on a patient's chest, as well as electrodes on each of thelimbs of the patient. The 12-lead ECGs are tools that may be foundfrequently in hospitals, typically in Intensive Care Units orHigh-dependency monitoring beds, as the output of a 12-lead ECG providesa more detailed look at three areas of the patient's heart—the anterior,lateral and inferior, and changes in certain segments of the ECG maysuggest an area of concern. However, the 12-lead ECG requires trainedpersonnel to accurately administer the 10 electrodes. Further, in orderto extract the more pertinent information from the 12-lead ECG,experienced and highly trained clinicians are required forinterpretation.

The above difficulties limit the usage of the 12-lead ECG as a triagetool in the ED. Prompt attention and assessment is usually required forincoming patients complaining of chest pain, and there typically may notbe availability of specially trained nurses and doctors to bothadminister the electrodes as well as to interpret the results. 12-leadECGs are intended for an in-depth “snapshot” view of a patient's cardiacsituation. Unfortunately, in the ED, such elaborate in-depth view maynot subject to a prompt interpretation, and as such, 3-lead ECGs aremuch more typically found in the EDs.

The applicants of the present invention had also previously developedmachine learning based risk assessment tools for identifying cardiacrisk in patients presenting to the ED, and incorporate by reference thespecifications identified as patent publication WO2011/115576.

DISCUSSION OF PRIOR ART DOCUMENTS

In the present embodiment, it is envisioned that a quick and accuratetriage tool be provided for use to assess presented patients complainingof chest pain, and operated by clinicians who can be presented with arisk score as assessment, and do not have to be highly experienced orhighly trained specialists.

System Architecture of Triage System of the Present Embodiment

FIG. 2 illustrates a system architecture of a triage system of thepresent embodiment. Triage system 100 is provided, which is designed tobe applied onto an incoming patient 102 presented to the ED andcomplaining of chest pains. The triage system 100 is intended to be aclinical tool for use in the ED, to assess incoming patients, and toprovide a risk score as an output. The risk score in such a systempertains to risk stratification of ACS in the patient, where perhaps alow score indicates that the chest pain is self-limited, while a highscore indicates the imminent possibility of cardiac arrest and/or lethalarrhythmias.

As an overview, the triage system 100 utilizes physiological and cardiacdata measurements, compiled with medical status information, andprocesses such inputs within an intelligent machine-learning scoringsystem which compares the present input to correlated past patientdiagnoses, in order to provide an insightful risk score as to the riskof ACS in the patient.

In the present embodiment, at least one medical status input 106pertaining to the incoming patient 102 is provided to the triage system100. In the embodiment, a computer interface system provided for an EDnurse to register the incoming patent 102, and to enter pertinentinformation relating to the medical status input 106 of the patient 102.The medical status input 106 is thereafter transmitted and logged into atriage system central processor 104, under an identifier for thepatient, typically by name or by identity registration number.

The medical status input 106 may be a medical history of the patient, adrug history, a drug allergy, a smoking history, a family history ofischemic heart disease, a record of angina events in the past 24 hours,an indication or description of current ailment or any other informationor factor which may be useful in the assessment or determination ofwhether the patient may suffer or may already be suffering from ACS. Itcan also include patient age, gender and demographic information.

Such medical status input 106 is typically taken through an oralinterview with the patient, but should a patient be somehow incapable ofregular conversation or does not have access to such pertinentinformation or be incapacitated in any way, it is envisioned that thecentral processor, is able to communicate with a centralized data server108 which has stored in memory such medical status input 106 of thepatient 102. The central processor 104 is thus capable of accessing thedata server 108, polling the data server 108 based on the relevantpatient identifier, retrieving the information required, and propagatingthe information into the present triage assessment. In the presentembodiment, the centralized data server 108 is a hospital data serverproviding and collecting information through various workstations in thehospital, and where the patient 102 has on previous occasion visited thehospital and provided information pertaining to his medical status andhistory. Alternatively, the centralized data server 108 may be aregional or national data server, where hospitals and medical carefacilities share patient information, subject of course to relevantprivacy laws and guidelines.

Physiological Measurements

The triage system 100 of the present embodiment also seeks to obtain asphysiological data input 110 from the patient 102 in the assessment ofrisk. Physiological data in the present case, may refer to a vital signdata of the patient 102. Vital sign data may be defined as clinicalmeasurements that indicate the state of a patient's essential bodyfunctions. For example, vital sign data may refer to a heart rate, arespiratory rate, a blood pressure reading, a temperature reading, aGlasgow Coma Score (GCS), an saturation of peripheral oxygen (SpO₂)reading, a pain score, or any other measurement or reading obtained fromthe patient which may be relevant in the assessment of ACS.

To obtain such physiological data 110, the above vital signs may bemeasured as follows. For example, the heart rate and the systolic anddiastolic components of a blood pressure reading may be measured using acombination blood pressure measurement device such as the Propaq CSVital Signs Monitor. Alternatively, devices such as a sphygmomanometeror a mercury manometer may be used to measure blood pressure. Heartrate, oxygen saturation reading and respiratory rate may be measuredusing a pneumogram. Separately, heart rate may be monitored with asimple pulse monitor, and oxygen saturation may be measured with a pulseoximeter. Measurement of SpO₂ may also be known as pulse oximetry, andas defined is the ratio of oxyhemoglobin to the total concentration ofhemoglobin present in the blood.

Glasgow coma scale (GCS) refers to the degree of spontaneity of thepatient's physical (such as limbs, eyes) motor and/or verbal response toinstructions from a medical professional. Pain score refers to thedegree of response (such as adduction, pronation or extension of a limbor body part; flexion or withdrawal) to pain applied to the patient.Tympanic (ear) temperature may be recorded using a tympanic thermometer.

Further, AVPU (“alert, voice, pain, unresponsive”) scores are recordedat triage and scored according to the best response during thecollection. A modified early warning score (MEWS) is also calculatedbased on collected data during presentation to the ED. MEWS provides asimple guide to determine illness of a patient, and is based on the fourphysiological readings (systolic blood pressure, heart rate, respiratoryrate, body temperature) and one observation (level of consciousness,AVPU).

Further, in the present embodiment, the presenting patient is also putthrough an immunoassay test to measure the presence of cardiac-TroponinT serum levels, which may infer cardiac conditions such as myocardialinfarction.

In the embodiment, the central processor 104 includes sufficient signalacquisition and processing capabilities to receive directly the signalinput from any or all of the physiological data inputs 110.Alternatively, the triage system 100 consolidates the measurement andobtaining of the above physiological inputs 110 in a self-containedphysiological data processing unit and thereafter provides digitalsignals to the central processor 104 for sorting and further processing.

Also, the central processor 104 may similarly access a centralized dataserver to obtain records of previously measured physiological data 110,and carry out a comparison with respect to present readings, to chartany differences in the patient's physiological state, and perhapsprovide an additional insight for a risk assessment.

12-lead ECG—Setup

In the present embodiment, a 12-lead ECG procedure is carried out by a12-lead ECG machine 112 or an ECG sensor to provide understanding as tothe electrical activity and function of the patient's 102 heart. In theembodiment, a Philips PageWriter TC series device is used. In order tocarry out the procedure, the attending ED nurse has to accurately attachthe ten electrodes required (refer to FIG. 1 for specific locations) inorder to adequately obtain electrical activity readings of the patient'sheart, which are thereafter interpreted into an ECG. Typically, ECGs areprovided as a continuous printout of the electrical signals as detectedby the attached electrodes. In the embodiment, a 5 minute ECG reading isused in the triage, but any other time period which allows for relevantreadings to be obtained may be similarly used.

In the present embodiment, the resultant ECG signals are provided to anECG post-processing module 114 provided as an application program on amemory module for operation on a processor on the 12-lead ECG machine112, the ECG post-processing module including a data acquisition device(DAQ) for receiving the analog sensor inputs. Sampling of the ECGsensors is carried out at 125 Hz. In the embodiment, the post-processingmodule 114 utilizes a LABVIEW interface embedded with MATLAB code fordata processing of the acquired electrical signals from the attachedelectrodes. The ECG is converted into a digital signal format to alloweasy manipulation and interpretation of the acquired electrode signalreadings. With regard to the term “module”, it is stated that the termrefers to the particular function or functions performed by theassociated processing unit; the “module” may or may not correspond toactual electrical circuitry.

The ECG post-processing module also operates a filter module (not shown)which processes the raw ECG data from the 12-lead ECG machine 112 tosuppress unwanted signals such as noise, motion, motion artifacts, powerline interference, and carries out any other manipulation necessary forthe accurate observation and interpretation of the ECG 120. A 5-28 Hzband-pass filter is used in the present embodiment, but other suitableconfigurations may also be possible.

Alternatively, the ECG signals obtained by the 12-lead ECG machine 112are directly provided to the central processor 104 and processed by anECG post processing module operating on the central processor 104.

In the present embodiment, an ECG extraction module 116 is provided foroperation within the framework of the ECG post-processing model 114. TheECG extraction module 116 is configured to act upon the digitized streamof electrical signal activity as obtained by the 12-lead ECG machine112. In addition, the ECG extraction module 116 acts upon the filtereddigitized stream of electrical signal activity as obtained by the12-lead ECG machine 112.

In the embodiment, the 12-lead ECG machine 112 also generates acontinuous hardcopy paper printout of the resultant ECG signals suchthat a trained medical practitioner may further analyze or cross-compareresults of the ECG with a risk score provided by the present triagesystem 100. The printout function could also be provided upon request bythe central processor 104, instead of a continuous hardcopy printout.

ECG—Interpretation

FIG. 3A illustrates a sample ECG printout 120 as provided by the triagesystem of the present embodiment. ECG 120 is a cutout of a continuousmonitoring that provides a snapshot view of the electrical activitysignal monitoring of a patient 102 as hooked onto the triage system 100.

A trove of information can be gleaned from the ECG 120 as to the healthand condition of a human heart. As mentioned above, not all medicalpersonnel working in the ED of a hospital are trained for such ECGinterpretation and analysis. The triage system 100 of the presentdisclosure provides an ECG extraction module 116 which analyzes the ECG120 and extracts parameters which provide such cardiac status insight.

The ECG 120 sets out the 12 leads or voltage difference measurementsbetween 2 electrodes. Essentially, the electrical activity of thefollowing leads are monitored and displayed as requested on the ECG 120:I, II, III, aVR, aVL, aVF, V1, V2, V3, V4, V5, V6. In this embodiment,the 12 leads are combined on to 3 concurrent trace lines. Alternatively,the 12 leads could be provided onto 12 separate trace lines.

Within an ECG tracing of the heartbeat, or the cardiac cycle, there isprovided a P wave, a QRS complex, a T wave, and a U wave. The baselinevoltage 121 of the electrocardiogram is known as the isoelectric line.Typically, the isoelectric line or the baseline voltage 121 is measuredas the portion of the tracing following the T wave and preceding thenext P wave. FIG. 3B illustrates a close up of the cardiac cycle 122from the ECG 120.

In medical definitions, the P wave 124 describes the main electricalvector directed from the sinoatrial node (SA node) towards theatrioventricular node (AV node), and spreads from the right atrium tothe left atrium. The QRS complex 126 represents a depolarization of theleft and right ventricles leading to a contraction of the heart. Theventricles have a larger muscle mass as compared to the atria, so theQRS complex 126 usually has a much larger amplitude than the P-wave.

In the present embodiment, a modified threshold-plus-derivative methodwas used to detect the QRS complexes, and all ectopics and othernon-sinus beats were excluded in accordance with the guidelines outlinedby the Task Force of the European Society of Cardiology and the NorthAmerican Society of Pacing and Electrophysiology.

FIG. 3C illustrates a close up of the QRS complex 126 of the cardiaccycle of FIG. 3B. The Q wave 128 is an initial negative deflection inthe QRS complex 126, and may or may not be present in a patient's ECGreadings. An abnormality in the Q wave 128 may indicate the presence ofan infarction in the patient 102. RR intervals were then calculatedbased on the sinus rhythm.

The R wave 130 is the initial upward deflection of the QRS complex 126,following the Q wave 128 in the ECG 120 and representing an earlydepolarization of the ventricles. The maximum amplitude of the R wave130 is referred to as the R peak amplitude, and is established by theamplitude of the R wave deflection as measured from the baseline, or theisoelectric line. A poor R wave progression may be attributed toanterior myocardial infarction, or could also be caused by left bundlebranch block, or Wolff-Parkinson-White syndrome, or right and leftventricular hypertrophy.

An RR interval 136 can be extracted from the ECG 120, which may bedefined as the interval between an R wave 130 and a subsequent R wave onthe ECG. The RR interval 138 is ECG-extracted interpretation of thepatient's heart rate.

S wave 132 is defined as the downward deflection of the QRS complex 126following the R wave in the ECG and represents a late depolarization ofthe ventricles. Also determinable from the QRS complex 126 is the QRSduration 134 which is defined as time period between the initialdeflection of the Q wave 128 and the end of the S wave 132 deflection. Aprolonged QRS duration 134 may indicate a hyperkalemia or a bundlebranch block.

A QRS amplitude can also be obtained from the QRS complex 126, where theamplitudes of the negative Q wave 128 and S wave 132 are subtracted fromthe amplitude of the positive R wave 130. An increased amplitude mayindicate cardiac hypertrophy. The ventricular activation time (VAT) 138,defined as the time duration between the beginning or the initialdeflection point of the Q wave 128 and the peak of the R wave 130, couldbe an indication of diastolic dysfunction.

A QRS axis can also be determined from the QRS complex 126, where theQRS axis is the averaged direction of electrical activity duringventricular depolarization, or the net vector of ventriculardepolarization. It can also be defined as the direction in which themean QRS current flows. The ECG extraction module 116 is able to processand indicate whether the QRS axis is in normal or in deviation towardsthe left or the right, which should be sufficient for clinicalinterpretation. Specific angle calculation could also be possible, ifnecessary.

Returning to FIG. 3B, T wave 140 occurs after the QRS complex 126 andrepresents the repolarization (or recovery) of the ventricles. Theinterval from the beginning of the QRS complex 126 to the apex of the Twave 140 is referred to as the absolute refractory period. The last halfof the T wave 140 is referred to as the relative refractory period (orvulnerable period). The T wave 140 can be described by its symmetry,skewness, slope of ascending and descending limbs, amplitude andsubintervals like the T_(peak)−T_(end) interval. T wave inversion, ornegative T waves, may be a sign of coronary ischemia, Wellens' syndrome,left ventricular hypertrophy, or central nervous system disorder.

The QT interval 142 is measured from the beginning of the QRS complex126, i.e. the initial negative deflection of the Q wave 128, to the endof the T wave 140. A prolonged QT interval 142 is a risk factor forventricular tachyarrhythmias and sudden death. The QT interval 142varies with heart rate—the faster the heart rate, the shorter the QTinterval, and for clinical relevance requires a correction for this,giving the QT interval correction (QTc). In the present embodiment,Bazett's formula is used in the calculation of the QTc, but othermethodologies may also be used. Bazett's formula is provided as follows:

$\begin{matrix}{{QT}_{C} = \frac{QT}{\sqrt{RR}}} & (1)\end{matrix}$

The ST segment 144 represents the connection between the QRS complex 126and the T wave 140. The ST segment 144 represents the period when theventricles are depolarized. Typically, the ST segment 144 is isoelectricand matched with the baseline. An ST elevation may be defined when theST segment 144 is abnormally high above the isolectric baseline 121. TheST elevation is obtained by measuring the vertical elevation between theECG trace of the ST segment 144 and the baseline 121, and may correspondto damage or pathlogical change to the cardiac muscle.

Presence of a left bundle branch block (LBBB), a cardiac conductionabnormality, is assessed for in the present embodiment. In a LBBB,activation of the left ventricle is delayed, which causes the leftventricle to contract later than the right ventricle. Criteria toobserve or assess a LBBB would include any or all of the following: Theheart rhythm must be supraventricular in origin; the QRS duration mustbe ≧120 ms; there should be a QS or rS complex in lead V1, there shouldbe a RsR′ wave in lead V6; the T wave 140 should be deflected oppositethe terminal deflection of the QRS complex 126. Some of the causes ofLBBB could be aortic stenosis, dilated cardiomyopathy, acute myocardialinfarction, or extensive coronary artery disease.

A right bundle branch block (RBBB) is also assessed, where in a RBBB,the right ventricle is not directly activated by impulses travellingthrough the right bundle branch. Criteria to observe or assess a RBBBwould include any or all of the following: The heart rhythm mustoriginate above the ventricles (i.e. SA, atria or AN) to activate theconduction system at the correct point; the QRS duration must be morethan 100 ms (incomplete block) or more than 120 ms (complete block);there should be a terminal R wave in lead V1 (e.g. R, rR′, rsR′, rSR′ orqR); there should be a slurred S wave in leads I and V6; the T wave 140should be deflected opposite the terminal deflection of the QRS complex126. An atrial septal defect is one possible cause of a RBBB.

In the embodiment, ECG extraction module 116 seeks to characterize anintraventricular conduction delay (IVCD). IVCD could be determined froma QRS duration 134 widening, where by a process of elimination, the QRSduration widening is caused by an IVCD if the manifestation is notcaused by a LBBB or a RBBB. IVCD may correspond to a myocardialinfarction, a cardiomyopathy with ventricular fibrosis, or a chamberenlargement.

Atrial abnormalities or atrial enlargements, atrial dilatations oratrial hypertrophy may also be detected in an ECG. Typically, suchabnormalities are found on the P wave 124, and in leads II, III, aVF orV1. In sinus rhythm, a right atrial depolarization wave precedes that ofthe left atrium and the combined depolarization wave is the P wave 124.

In a right atrial abnormality, the right atrial depolarization lastslonger than normal and its wave extends to into the left atrialdepolarization. Although the amplitude of the right atrialdepolarization current remains unchanged, its peak now falls on top ofthat of the left atrial depolarization wave. As a result, the combinedthe P wave, is taller than normal but its width remains.

In a left atrial abnormality, the left atrial depolarization lastslonger than normal but its amplitude remains unchanged. Therefore, theheight of the resultant P wave remains within normal limits but itsduration is extended. A notch (broken line) near its peak may or may notbe present. More quantitative analysis may also be provided to obtain aleft atrial abnormality and a right atrial abnormality.

Ventricular hypertrophy (VH) is the thickening of the ventricular walls(lower chambers) in the heart. Although left ventricular hypertrophy(LVH) is more common, enlargement can also occur in the right ventricle,or both ventricles. While ventricular hypertrophy occurs naturally as areaction to aerobic exercise and strength training, it is mostfrequently referred to as a pathological reaction to cardiovasculardisease, or high blood pressure.

In the embodiment, the Sokolow-Lyon index is used to diagnose LVH in theECG, although the accuracy of diagnoses is increased with the use ofmultiple criteria sets. In Sokolow-Lyon, the criteria for diagnosis isfor S in V1+R in V5 or V6 (whichever is larger) mm; and R in aVL≧11 mm.Causes of increased afterload that can cause LVH include aorticstenosis, aortic insufficiency and hypertension. Primary disease of themuscle of the heart that cause LVH are known as hypertrophiccardiomyopathies, which can lead into heart failure. Long-standingmitral insufficiency also leads to LVH as a compensatory mechanism.

In right ventricular hypertrophy, (RVH), conditions occur which decreasepulmonary circulation, meaning blood does not flow well from the heartto the lungs, and extra stress can be placed on the right ventricle. AnECG with right ventricular hypertrophy may or may not show a right axisdeviation on the ECG. Certain criteria for assessment may include anyone or more of the following: right axis deviation (>90 degrees) inpresence of disease capable of causing RVH; R in aVR>5 mm; R in aVR>Q inaVR; any one of the following in lead V1: R/S ratio>1 and negative Twave; qR pattern; R>6 mm, or S<2 mm, or rSR′ with R′>10 mm.

Atrial fibrillation (AF) is the most common cardiac arrhythmia(irregular heart beat). In AF, the normal regular electrical impulsesgenerated by the sinoatrial node are overwhelmed by disorganizedelectrical impulses usually originating in the roots of the pulmonaryveins, leading to irregular conduction of impulses to the ventricleswhich generate the heartbeat. AF may occur in episodes lasting fromminutes to days (“paroxysmal”), or be permanent in nature. A number ofmedical conditions increase the risk of AF, particularly mitral stenosis(narrowing of the mitral valve of the heart).

In the diagnosis of AF, characteristic would be the absence of P waves,with disorganized electrical activity in their place, and irregular R-Rintervals due to irregular conduction of impulses to the ventricles.

Other characteristics or parameters which can aid with the diagnosis orassessment of a risk score as to determining ACS for a patient can alsobe extracted from the 12-lead ECG and provided for analysis.

HRV

In the embodiment, a heart rate variability (HRV) extraction module 150is provided in the ECG post-processing module 114 and functions toextract parameters related to HRV from the ECG 120.

As mentioned above, several systems and methodologies seek to providesuch input for risk stratification utilizing a plurality of parametersfor such determination. In addition to the various clinical factors aspresented above, HRV is a potentially useful approach that can beapplied at the point of clinical assessment.

HRV is the physiological phenomenon of variation in the time intervalbetween heartbeats and is defined as variation in the beat-to-beatinterval. HRV is sometimes also known as RR interval 136 variability.The R-wave of a particular heartbeat corresponds to the point in thecardiac cycle of the early systolic phase, and from a signal processingpoint of view, provides a reliable time-fiducial for making cardiaccycle interval measurements.

Variation in the beat-to-beat interval is a physiological phenomenon.The SA node of the heart receives several different inputs and theinstantaneous heart rate or RR interval and its variation are theresults of these inputs. HRV is affected by the autonomic nervoussystem, which consists of the sympathetic nervous system (SNS) and theparasympathetic nervous system (PSNS), and which are also inputs to theSA node.

Observed HRV is believed to be an indicator of the dynamic interactionand balance between the SNS and PNS, providing a measure of nervoussystem competence. HRV serves as an indicator for the diagnosis andassessment of a variety of conditions that are affected by the autonomicsystem ranging from congestive heart failure to sleep apnoea. Forexample, reduced HRV is sometimes believed to be an independentpredictor of cardiac death and mortality after myocardial infarction inpatients. Reduced HRV is also seen after sudden cardiac arrest and inpatients with diseases such as diabetes, uraemia and hypertension.

HRV Parameters

From detected QRS complexes 126 in the ECG 120, the processed RRintervals 136 can be obtained. The RR intervals 136 are used tocalculate the following HRV parameters, from which include time domainand frequency domain analyses.

Examples of Time Domain Measures are:

Time Domain Measures

1. Average length of the RR interval (aRR): Mean of all sinus RRintervals (N-N) in sequence

2. Standard deviation of all N-N interval (SDNN)

3. Mean heart rate (mean HR)

4. Standard deviation of all instantaneous heart rate values (SDHR)

5. NN50 (count): Number of consecutive RR intervals differing by morethan 50 ms

6. pNN50(%): Percentage of consecutive RR intervals differing by morethan 50 ms

7. HRV triangular index: Total number of all N-N intervals divided bythe height of the histogram of all NN intervals.

8. Baseline width of a triangle fit into the N-N interval histogramusing a least squares technique (TINN)

9. Square root of the mean squared differences of successive N-Nintervals (RMSSD): The square root of the mean of the sum of the squaresof differences between adjacent N-N intervals.

Frequency Domain Measures

Frequency domain measures are calculated based on the power spectrum ofthe RRI sequence which is generated using a Lomb-Scargle periodogram.The following parameters are then calculated:

1. Total power (TP) (ms²): Variance of N-N intervals over the segmenttill 0.4 Hz

2. VLF (ms²): Power in very low frequency range (<0.04 Hz)

3. LF (ms²): Power in low frequency range (0.04-0.15 Hz)

4. HF (ms²): Power in high frequency range. (0.15-0.4 Hz)

5. LF norm (nu): LF power in normalized units: LF norm=LF/(TP−VLF)×100%

6. HF norm (nu): HF power in normalized units: HF norm=HF/(TP−VLF)×100%

7. LF/HF: Ratio of LF/HF

The extracted parameters are thereafter provided to the centralprocessing unit 104 for further processing.

In summary, the triage system 100 utilizes the above information andparameters obtained through medical status input 106, the physiologicaldata input 110, the 12-lead ECG machine 112 and subsequently the ECGextraction module 116 and the HRV extraction module 150, to determinerelevant weighted classifiers, which relate to the importance of theinformation and/or parameter to the determination of ACS.

Previous Patient Data Collection

Further provided in the triage system 100 of the present embodiment, isan access to a database of accumulated past patient data. Such adatabase is hosted on a centralized data server 108, for which thecentral processor 104 of the triage system 100 has data access to. Anensemble-based scoring system 160 is provided and operates on thecentral processor 104 which utilizes the accumulated past patient datain training up a machine learning structure, on which reliable decisionscan be expected.

In the present disclosure, past patient data is collated from patientspresented to the ED of a hospital with undifferentiated andnon-traumatic chest pain. Patients in non-sinus rhythm (e.g. asystole,supraventricular and ventricular arrhythmias, complete heart block) andpatients who were discharged against medical advice or transferred toanother hospital within 72 hours of arrival at the ED were excluded. Aneligible patient who arrived at the ED with chest pain was randomlyscreened and recruited by trained medical personnel.

The outcome of the study was a compilation of severe complicationswithin 72 hours of arrival at the ED, extracted from the electronichospital records. Patients were considered to have met the outcome ifthey had at least one of the following severe complications: all-causemortality, cardiac arrest, sustained ventricular tachycardia (VT),hypotension requiring inotropes or intra-aortic balloon pump (IABP)insertion, intubation or mechanical ventilation, complete heart block,and bradycardia requiring insertion of a pacing wire.

Parameters of data which are recorded and extracted by the presenttriage system 100 include any or all of the parameters listed above, aswell as any other not listed, but which may assist in the assessment ofACS. Further, the outcomes of the past patients from the visit to the EDare also utilized, which provides the reference for the various dataparameters.

With the advancement of computational techniques, machine learning hasbeen found to be useful for scoring systems to improve predictiveperformance, handle imbalanced data and enhance system adaptability. Inthe present disclosure, the ensemble-based scoring system 160 isprovided where an intention is to provide an intelligent scoring systemin combining HRV and 12-lead ECG parameters and vital signs to predictacute cardiac complications within 72 hours among critically illpatients presented with chest pain. The scoring system 160 of thepresent embodiment utilizes a unique machine learning structure, withwhich reliable decisions can be expected.

Upon collating of past patient data, and investigating such data, it wasfound that the outcome distribution is highly imbalanced. Imbalanceddata is defined where there exists a majority class with normal data anda minority class with abnormal data, in this case, patients with acutecardiac complications within 72 hours. When applying machine learningalgorithms on such an imbalanced dataset, the majority class willdominate the learning process and consequently results in poorgeneralization performance on unknown testing samples. Typical solutionsto handle imbalanced data include under-sampling majority classes andover-sampling minority classes. However, the prevalence rate observedfrom the resultant pilot data is fairly low (<5%). As a result, neitherstate-of-the-art classification techniques nor conventional imbalancehandling strategies may be able to provide satisfactory predictionresults. In order to provide reliable prognosis with HRV parameters,12-lead ECG parameters and vital signs, a learning framework tailoredspecifically for imbalanced data is important and will serve as a majorfactor in controlling system performance.

The applicants have previously proposed a geometric distance basedscoring system in which inputs were continuous variables, which may beapplicable to parameters such as heart rate, blood pressure, orrespiratory rate amongst others under physiological data inputs and evenHRV parameters as extracted from the ECG. In the present disclosure, itis intended for 12-lead ECG parameters to be integrated into the scoringmodel; however, these measurements are in discrete format, i.e., either0 or 1. As a result, a new scoring system is required, which is able tohandle both continuous and discrete variables as inputs.

Prior to calculation of the machine learning score, the original inputsshould be classified into a [−1, 1] interval by performing a min-maxnormalization. Given a dataset X=[x₁, x₂, . . . , x_(K)] where each xrepresents a patient, min_(A) and max_(A) are then defined to denote theminimum and maximum values of an attribute vector A=[x₁(m), . . . ,x_(K)(m)], where m is the number of features (in the embodiment, m isthe total number of 12-lead ECG parameters, HRV parameters and vitalsigns being utilized as classifiers) and K is the total number of pastpatient data samples. Min-max normalization maps a value, v, of A to vin the range [min_(A), maxA_(A)] by computing the equation:

$\begin{matrix}{v^{\prime} = {{\frac{v - \min_{A}}{\max_{A}{- \min_{A}}}( {\max_{A}^{\prime}{- \min_{A}^{\prime}}} )} + \min_{A}^{\prime}}} & (2)\end{matrix}$

It is noted that the normalization process is able to preserve therelationships among the original data values, which thereforefacilitates the machine learning based risk score prediction in theensemble-based scoring system 160.

Ensemble-Based Decision Making

In the medical community, it is good practice for medical practitionersto seek a second or further opinion before making a final decision onthe situation. By consulting several experts with various backgrounds,medical practitioners can weigh their suggestions or pick up the mostinformed one. For example, the suggestion by a senior clinician could begiven a higher weight than that of a junior clinician. In critically illcases, final decisions may be given by a committee of experts in adiscussion and outcome even put to a vote. Given a desire to operate asclosely as possible to real-world situations, computational intelligencemethods seek to simulate the process of decision from multiple experts.Such intelligent learning systems have various names such as ensemblelearning systems, mixture of experts, and multiple classifier systems.

The principle behind these techniques is to discover an optimal way tocombine the suggestions of individual experts so as to achieve areliable final decision. FIG. 4A illustrates a general structure ofensemble learning based system. Within this structure, each individualexpert may be referred to as a classifier 164. In FIG. 4A, classifier164 ₁ references classifier 1, classifier 164 _(t) references classifiert, and classifier 164 _(T) references classifier T. Similar features ofsimilar functions are similarly numbered. In the present embodiment,each expert also refers to a parameter which may provide an insight tothe assessment of ACS in a patient. Each ensemble classifier is provideda weight to represent the importance of that classifier. In the presentdisclosure, the weight is determined by the contribution of itscorresponding classifier and is derived from training process, i.e. theweightage of the classifier is related to the relevance of thatparticular parameter in the assessment of ACS.

Typical ensemble learning methods usually generate a predictive labelrather than a score as the output. However, and as mentioned previously,a risk score is more informative than a class label to clinicians inproviding insight for decision making. In the present disclosure, asimple ensemble score or risk score 168 is provided as the predictionoutput of the system.

In the embodiment, access is from the triage system 100 and theensemble-based scoring system 160 to a training dataset 162. Trainingdataset 162 is also referenced by L and includes K samples (x_(k),y_(k)) where k=1, 2, . . . , K and y_(k) is the class label. Given anincoming testing sample x from testing data 166 obtained from thecentral processor 104 of the triage system 100 as applied to an incomingpatient, its label y can be predicted by a single classifier φ(x, L)where the class label is either C₀ or C₁. In the present disclosure,label C₀ indicates that the patient is normal (a negative ACS outcome)while label C₁ indicates that the patient has acute cardiaccomplications within 72 hours (a positive ACS outcome). As illustratedin FIG. 4A, a set of T independent classifiers can be derived from inputparameters, and their corresponding weights can be determined from asample training data set 162. The risk score 168 on sample x iscalculated using the equation as follows:

$\begin{matrix}{{RS}_{x} = {\frac{\sum\limits_{y \in C_{1}}\; {{\phi_{t}( {x,L} )} \cdot w_{t}}}{{\sum\limits_{y \in C_{1}}\; {{\phi_{t}( {x,L} )} \cdot w_{t}}} + {\sum\limits_{y \in C_{0}}\; {( {1 - {\phi_{t}( {x,L} )}} ) \cdot w_{t}}}} \times 100}} & (3)\end{matrix}$

where the output of classifier φ_(t)(x, L) is either 0 or 1 and itscorresponding predicted label y is C₀ or C₁, respectively.

The risk score is based on the measurements of weighted positiveprediction and weighted negative prediction. The weighted positiveprediction is defined as the sum of weights whose correspondingclassifiers predict a label of C₁ while the weighted negative predictionis defined as the sum of weights whose corresponding classifiers predicta label of C₀ on testing sample x. The principle behind the learningmachine of the ensemble-based scoring system 160 is an attempt tosimulate the process of real-world decision making. Since the presentdisclosure addresses the parameters and classifier analysis as binaryclass problems, the presentation of risk score calculation can besimplified in accordance with the following equation.

$\begin{matrix}{{RS}_{x} = {\frac{\sum\limits_{t = 1}^{T}\; {{\phi_{t}( {x,L} )} \cdot w_{t}}}{\sum\limits_{t = 1}^{T}\; w_{t}} \times 100}} & (4)\end{matrix}$

Further, in the present disclosure, it is desired to determine how toselect suitable individual classifiers to create a decision ensemble andalso to determine a useful methodology for decision combinations.Addressing the above is difficult for most medical scenarios wheredatabases are usually imbalanced, i.e., positive samples are much lessthan negative samples. For example in predicting acute cardiaccomplications, there are less than 5% positive samples amongst the pastpatient data. In this present disclosure, two embodiments for theensemble-based scoring system are proposed—a first system adapting anunder-sampling method to calculate a risk score, and a second systemincluding a hybrid-sampling algorithm.

Ensemble-Based Scoring System 1—USS

FIG. 4B illustrates a first scoring system utilizing an under-samplingmethod to calculate a risk score. In the embodiment, the ensemble-basedscoring system 160 is an under-sampling based scoring system (USS) 161where an under-sampling technique is applied. The USS is designed toconduct a risk score prediction on an imbalanced dataset. As a providedinput, training dataset 163 (or L) provides a set of minority classsamples P and a set of majority class samples N, and also a determinednumber of individual classifiers T. t is determined to be a reference tothe determined classifiers 165 _(t), where t=1, . . . , T.

In the embodiment, the under-sampling method randomly samples a subsetN_(t) from N where |N_(t)|<|N|. In most medical scenarios, |P|<<|N| suchthat |N_(t)|=|P| is selected, where P represents a set of samples withpositive outcomes and N represents a set of samples with negativeoutcomes. The resultant samples are combined to provide a balanceddataset S, where S_(t)=P+N_(t).

Thereafter, the USS randomly samples T subsets and trains T independentclassifiers with N_(t) and P for each classifier S_(t). A classificationmodel S_(t) is thereafter built. More detailed information on building asuitable classification model can be referenced from the articleentitled “An Intelligent Scoring System and Its Application to CardiacArrest Prediction” (Nan LIU et al, Nov. 2012, IEEE Transactions onInformation Technology in Biomedicine) by an inventor as to the presentdisclosure, the article being incorporated fully by reference in thispresent disclosure. This classification model S_(t) is then applied ontothe incoming testing sample x from testing data 166 to produce aprediction output φ_(t)(x, S_(t)) that is either 0 or 1.

In the present embodiment, it is assumed that all T individualclassifiers equally contribute to the decision making and the weightagevalue of the classifier φ_(t) has a w_(t) value as set to 1. In otherembodiments, the weightage value of each classifier φ_(t) may beevaluated and thereafter provided with a value which reflects theimportance of the parameter or classifier towards the assessment of ACSin a patient.

Several other state of the art ensemble learning methods combine theoutputs of all classifiers into one composite prediction. However, inthe present disclosure, it is provided where the total number ofpositive predictions, as well as negative predictions are calculated,and equation (4) is used to estimate a risk score, i.e. a risk score ispredicted for x with: RS_(x)=(Σ_(t=1) ^(T)φ_(t)(x,S_(t))·w_(t)/Σ_(t=1)^(T)=w_(t))×100. This risk score is provided as an output or ensemblescore 169 of the USS 161.

In this present disclosure, a support vector machine (SVM) is providedas an individual classifier in the ensemble learning based scoringsystem. SVM implements a conceptually simple idea, i.e. input vectorsare non-linearly mapped to a high-dimensional feature space in which alinear decision hyperplane is constructed to separate input vectors withmaximum margin.

FIG. 5A is a block diagram 500 of an algorithm of USS 161. In a firstprocess block 502 of the present embodiment, t is determined to be areference to the determined classifiers 165 _(t), where t=1, . . . , T.In a second process block 504, a balanced dataset S_(t) is first createdby combining P and N_(t), where N_(t) is randomly sampled from N, andwhere P and N_(t) have the same number of samples.

In a next process block 506, a classification model φ_(t) is built basedon S_(t). In process block 508, it is assumed that all T individualclassifiers are equally contributing to the decision making and thevalue of w_(t) is set to 1.

Process blocks 502 to 508 are then repeated to obtain all classificationmodels φ_(t), and corresponding to them each w_(t) set to 1. In aprocess block 510, the classification models are applied to incomingtesting sample x to obtain prediction outputs φ_(t)(x, S_(t)) that areeither 0 or 1.

In a following block 512, a risk score is predicted for incoming testingdata x with equation (4): RS_(x)=(Σ_(t=1)^(T)φ_(t)(x,S_(t))·w_(t)/Σ_(t=1) ^(T)=w_(t))×100. This risk score isprovided as an output or ensemble score 169 of the USS 161.

Ensemble-Based Scoring System 2—HSS

FIG. 4C illustrates a second scoring system utilizing a hybrid-samplingapproach according to a second embodiment. In the above USS 161, randomunder-sampling is used for subset selection in the majority classsamplings. The selection process utilized in system USS 161 provides anunsupervised strategy to explore a majority class of the data samples,i.e., the performance of each individual classifier may not bedeterminable, even though some of the individual classifiers maycontribute less to the decision ensemble. Therefore, in a secondembodiment, there is provided a supervised strategy for individualclassifier selection such that a robust decision ensemble with strongdiscriminatory power can be built.

In the embodiment, the ensemble-based scoring system is ahybrid-sampling based scoring system (HSS) 260 where both under-samplingand over-sampling techniques are applied. State of the art intelligentmachine learning systems typically utilize over-sampled data to enhancetraining. In this present embodiment, an over-sampling technique is usedto generate synthetic data for validating individual classifiers, so asto provide a hierarchy for selecting the more relevant classifiers tocreate the decision ensemble.

Input is similarly provided, where training dataset 262 provides a setof minority class samples P and a set of majority class samples N, where|P|<<|N|. There is also a determined number of individual classifiers T,and a number of individual classifiers for optimization J. T and J areindependent variables. T defines the ensemble size and J defines thenumber of classifiers for optimization. Each classifier is chosen out ofJ classifiers to create an ensemble containing T classifiers.

FIG. 5B is a block diagram 550 of an algorithm of HSS 260. In a firstprocess block 552 of the present embodiment, t is determined to be areference to the determined classifiers 264 _(t), where t=1, . . . T,and j is determined to be a reference to the determined optimizedclassifiers 265, where j=1, . . . , J. In a second process block 554, abalanced dataset S_(tj) is first created by combining P and N_(tj),where N_(tj) is randomly sampled from N, and where P and N_(tj) have thesame number of samples.

In a following process block 556, a synthetic dataset S_(tj) is createdby combining P and N_(tj), where P is obtained by applying SMOTE on Pand N_(tj) is obtained by applying SMOTE on N_(tj). SMOTE stands forSynthetic Minority Over-sampling Technique, and is used in the art forthe construction of classifiers from imbalanced datasets.

In SMOTE, the class data set, typically the minority class, isover-sampled by taking each data sample and introducing syntheticexamples along a line segment joining any/all of a k sample classnearest neighbors, where k is a predetermined variable based on theamount of over-sampling required. For example, if the amount ofover-sampling required is 200%, only 2 neighbors from the 5 nearestneighbors are chosen and one sample is generated in the direction ofeach. In generating the synthetic sample, a random number is providedbetween 0 and 1, which is thereafter multiplied with and added to thesample vector under consideration. This provides a selection of a randompoint along the line segment between two specific features, andeffectively allows the decision region of the class data set to becomemore general, typically in discussion of the minority class incomparison with the majority class.

SMOTE provides the addition of synthetic samples which cause theclassifiers to create larger and less specific decision regions, ratherthan smaller and more specific region. The overall result is such thatdecision trees generalize better. In the embodiment, both data sets Pand N_(tj), where both sets include a number of samples equal to theminority class set, are applied with SMOTE to obtain a new syntheticdataset S_(tj), where S_(tj)=P+N_(tj).

In a next process block 558, a classification model φ_(tj) is builtbased on S_(tj), and the trained model φ_(tj) is validated based onS_(tj). The resultant prediction accuracy of the classification modelφ_(tj) is stored as Acc_(tj).

Process blocks 552 to 558 are then repeated to obtain the predictionaccuracy of each classification model, until all J classifiers foroptimization have been processed.

In a process block 560, the dataset S_(tj) with the highest predictionaccuracy Acc_(tj) is selected as the first balanced dataset as S₁ andits weightage w₁ is set as corresponding prediction accuracy Acc_(tj).

In a following process block 562, a classification model φ₁ isbuilt/trained based on S₁ and the trained classification model φ₁ isapplied to incoming testing sample x to produce a prediction outputφ₁(x, S₁) that is either 0 or 1. In another embodiment, the previouslybuilt classification model for which the prediction accuracy wasobtained for is reused.

Process blocks 552 to 562 are then repeated to obtain the predictionoutput of each weighted and sorted classification models φ_(t), untilall T classifiers have been processed. In particular, the balanceddatasets are sorted in the order S₁, S₂, . . . , S_(t), . . . , S_(T),each with a corresponding weightage w₁, w₂, . . . w_(t), . . . w_(T),based on the earlier obtained prediction accuracy. The correspondingclassification models φ₁, φ₂, . . . , φ_(t), . . . , φ_(T), are builtbased on the balanced datasets and thereafter applied to incomingtesting sample x to obtain prediction outputs φ_(t)(x, S_(t)) that areeither 0 or 1 in a process block 564.

In a process block 566, a risk score is predicted for incoming testingdata x with equation (4): RS_(x)=(Σ_(t=1)^(T)φ_(t)(x,S_(t))·w_(t)/Σ_(t=1) ^(T)=w_(t))×100. This risk score isprovided as an output or ensemble score 268 of the HSS 260.

To further describe the embodiment, a weighted decision ensemble can becreated to predict risk scores on incoming testing sample x. Anadvantage of the HSS 260 is its unique strategy for classifierselection, which is a supervised process that takes the performance ofan individual classifier into account.

As mentioned previously, there is an intention to introduce analternative way of using over-sampled data, i.e., for validation insteadof training. In the HSS 260, a classifier 264 ₁ is trained with S_(tj)and validated with S_(tj), and the validation accuracy Acc_(tj) isrecorded as a weightage to indicate the importance of the classifier. Ahigher Acc_(tj) value indicates that the individual classifier φ_(tj)would be able to contribute more to the decision ensemble. Thus, thedataset S_(tj) with the highest Acc_(tj) is selected and sorted in orderas S₁, S₂, . . . , S_(t), . . . , S_(T), with a corresponding weightagewt set with the previously obtained validation accuracy Acc_(tj) as itsvalue. The classifier φ_(t)(x, S_(t)) is then built or trained forensemble creation, and derivation of the ensemble score is similarlyprovided from equation (4).

Several advantages of utilizing HSS 260 as the ensemble-based scoringsystem of the present disclosure have been identified, and are providedas follows.

The present disclosure utilizes at least or up to 13 parameters derivedfrom the 12-lead ECG, at least or up to 16 HRV parameters and at leastor up to 8 vital signs for a risk score prediction. These parametershave been selected based on an understanding that 12-lead ECG parameterscombined with HRV parameters would be able to provide a more accurateprediction as compared to a scoring system based on a single type offeature. Furthermore, the proposed HSS 260 as the ensemble-based scoringsystem 160 is flexible as applied to real-world scenarios; it is notlimited to above-mentioned parameters as inputs. Based on clinicalneeds, any number of parameters can be fed into the scoring system HSS260 for risk prediction if they are found to be able to achieve requiredprediction performance.

However, to accommodate various types of inputs, the system needs to beretrained. The present HSS 260 provides a flexibility in retraining,without compromising the quality of the risk score output. A retrainingwould take into account each classifier, validates the classificationmodel for each classifier and sorts each classifier, including the newparameter/classifier based on the validation accuracy or relevantimportance of each parameter/classifier.

The proposed scoring system HSS 260 also has the flexibility of modelretraining to handle different conditions such as changes of inputfeatures and changes of prediction targets. The scoring system adopts amachine learning structure, and thus includes an effective modeltraining strategy to deal with potential changes within a trained model.For example, if a subset of features is preferred in risk prediction,simply re-running the proposed scoring system with these new featureswould then generate an entirely new model. A training requirement wouldof course be that provided training samples from accumulated pastpatient data and incoming testing samples must have the same inputparameters for assessment. Having the flexibility of retraining, theproposed system can be easily implemented regardless of diseases anddemographics. As long as clinically meaningful predictors are collected,any disease under any demographics can be predicted. Satisfactoryperformance could be achieved given that the study is well designed andthe collected data is accurate. For example, a well-trained predictionmodel for Asian population may not work well in American population.

Applicable to both USS 161 and HSS 260, the pre-assessment learning tobuild up classification models for risk score prediction allows thescoring system 160 and thus the triage system 100 to be free fromadditional processing load, for example, if the scoring system requireda live identification of relevant input parameter and constant access tothe past patient data. In the present disclosure, the triage system 100can be a standalone system, without need for access to a centralizeddata server 108 to obtain past patient data. There could be multipleadvantages of such a feature, for example, in providing a triage systemfor use in a mass casualty event, or even in mobile hospitals operatingwithout standard emergency department resource allocation.

Central Processing—USS

FIG. 6 illustrates a modular layout of an ensemble-based scoring systemof the triage system according to an embodiment. In the embodiment, USS161 is provided as the ensemble-based scoring system, and is designed tooperate as an application program based in a memory module of thecentral processor 104, and operating on the central processor 104. Asindicated, “module” as described refers to the particular function orfunctions performed by the associated processing unit; and the “module”may or may not correspond to actual electrical circuitry.

In general, the USS 161 includes 3 larger functional modules asindicated in the modular layout 600—a data acquisition and processingmodule 602, a classifier generation module 604, and an analysis module606. Functionally, the data acquisition and processing module 602supervises the request and receipt of accumulated past patient datathrough a data access module 608 from a database 610 that, in thepresent embodiment, is hosted on a centralized data server 108, andapart from the central processor 104 of the triage system 100. In anembodiment, the provided data communication link between data accessmodule 608 and the database 610 is a wireless communication link, forexample in accordance with WLAN protocol IEEE 802.11a-ad. An on-demandaccess is provided for communication between the data access module 608and the database 610, but a perpetual access situation can also beprovided.

Data access module 608 obtains from the database 610 informationrelating to previous patients, in particular, groups of data as sortedby a unique identifier, in this case, a patient name or an identityregistration number. Critically, the outcome of the patient's visit tothe ED is also included and obtained. The data groups obtain include aplurality of various parameters relating to the functional purpose ofthe scoring system 160, the assessment of a risk score in relation tothe risk of ACS of an incoming patient. The parameters as obtained maybe the parameters listed in the above description, that of medicalstatus, physiological data, ECG parameters, and HRV parameters.Alternatively, other parameters as decided as necessary may be obtained.

After past patient data has been obtained by the data access module 608,the data is provided onto a sorting module 612 for further processing.Sorting module 612 separates each parameter for assessment, as decidedby the triage system 100, from the group of data as uniquely identified,and puts them into parametrically sorted datasets. The sorting module612 further tags each processed parameter with the unique identifier ofeach patient it belongs to, as well as the outcome of the patient's tripto the hospital. Typically, the resultant parametrically sorted datasetsare imbalanced datasets, where there exists a majority class with normaldata, in this case a negative outcome, and a minority class withabnormal data, in this case a positive outcome.

After sorting, the sorted data is provided to a sampling module 614,wherein the sampling module 614 identifies a majority class or datasetof the provided imbalanced dataset, the majority class including a firstnumber of data samples, and a minority class or dataset including asecond smaller number of data samples. The sampling module 614thereafter extracts a subset of a third number of data samples from themajority data set, such that the number of samples of the majority datasubset is equal to that of the minority data dataset. This provides forbalanced datasets of parameters relating to classifiers. In theembodiment, the extraction of the majority data subset is entirelyrandom.

The balanced datasets are thereafter provided to a training module 616,where data received by the USS 161 is trained into classifiers such thatincoming patient data may be received and evaluated against the trainedclassifier to provide a portion of a risk assessment score as to thehealth of a patient. In training module 616, the balanced dataset isused to build a classification model, where the classification model, orclassifier acts as an expert in providing an insight on whether theincoming patient is likely to have a positive outcome in relation toACS.

In the present embodiment, after the classification models are built bythe training module 616, the models are provided to a weighing module618. In the present embodiment, it is assumed that all of theclassifiers provide an equal contribution to the decision making and assuch, the weighing module 618 provides all the classifiers with an equalweight of 1. In other embodiments, the weightage value of eachclassifier may be evaluated and thereafter provided with a value whichreflects the importance of the parameter or classifier towards theassessment of ACS in a patient. This value would then be provided to theclassifier by the weighing module 618.

An output of the classifier generation module 604 is thus that oftrained classifiers 620, related to the various parameters indicated asbeing of importance toward the assessment of a risk score by the triagesystem 100. Preferably, the classifiers 620 are sorted according totheir weight or importance. However, in this embodiment, this is notcarried out as the weightage of each classifier has been set to 1.

The USS 161 is now ready for usage and assessment of an incomingpatient, and for providing a risk score therefrom. Analysis module 606includes a testing module 622 arranged to receive incoming patient testdata 624 from the central processor 104 of the present triage system100. In this embodiment, each received parameter of the incoming patienttest data 624 is tested with a corresponding trained classifier 620 toprovide a prediction output of whether the patient is likely to achievea negative or positive outcome. The prediction output is provided in abinary output format of 0 or 1.

After testing and evaluation of all the parameters and classifiers, theresultant prediction outputs are collated and passed on to a scoringmodule 626. Scoring module 626 calculates the risk score for theincoming patient, based on his incoming measurement parameters 624, as anormalized summation of the binary prediction outputs of all theweighted classifiers. This generated risk score provides the medicalpersonnel using the triage system 100 with a calibrated insight as towhether the incoming patient is at a risk of ACS.

Central Processing—HSS

FIG. 7 illustrates a modular layout of an ensemble-based scoring systemof the triage system according to a second embodiment. In theembodiment, the HSS 260 is provided as the ensemble-based scoringsystem, and is designed to operate as an application program based in amemory module of the central processor 104, and operating on the centralprocessor 104. As indicated, “module” as described refers to theparticular function or functions performed by the associated processingunit; and the “module” may or may not correspond to actual electricalcircuitry.

In general, the HSS 260 includes 3 larger functional modules asindicated in the modular layout 700—a data acquisition and processingmodule 702, a classifier generation module 704, and an analysis module706. Functionally, the data acquisition and processing module 702supervises the request and receipt of accumulated past patient datathrough a data access module 708 from a database 710 that, in thepresent embodiment, is hosted on a centralized data server 108, andapart from the central processor 104 of the triage system 100. In anembodiment, the provided link between data access module 708 and thedatabase 710 is a wireless communication link, for example in accordancewith WLAN protocol IEEE 802.11a-ad.

Data access module 708 obtains from the database 710 informationrelating to previous patients, in particular, groups of data as sortedby a unique identifier, in this case, a patient name or an identityregistration number. Critically, the outcome of the patient's visit tothe ED is also included and obtained. The data groups obtain include aplurality of various parameters relating to the functional purpose ofthe scoring system 160, the assessment of a risk score in relation tothe risk of ACS of an incoming patient. The parameters as obtained maybe the parameters listed in the above description, that of medicalstatus, physiological data, ECG parameters, and HRV parameters.Alternatively, other parameters as decided as necessary may be obtained.

After past patient data has been obtained by the data access module 708,the data is provided onto a sorting module 712 for further processing.Sorting module 712 separates each parameter for assessment, as decidedby the triage system 100, from the group of data as uniquely identified,and puts them into parametrically sorted datasets. The sorting module712 further tags each processed parameter with the unique identifier ofeach patient it belongs to, as well as the outcome of the patient's tripto the hospital. Typically, the resultant parametrically sorted datasetsare imbalanced datasets, where there exists a majority class with normaldata, in this case a negative outcome, and a minority class withabnormal data, in this case a positive outcome.

After sorting, the sorted data is provided to a sampling module 714,wherein the sampling module 714 identifies a majority class or datasetof the provided imbalanced dataset, the majority class including a firstnumber of data samples, and a minority class or dataset including asecond smaller number of data samples. The sampling module 714thereafter extracts a subset of a third number of data samples from themajority data set, such that the number of samples of the majority datasubset is equal to that of the minority data dataset. This provides forbalanced datasets of parameters relating to classifiers. In theembodiment, the extraction of the majority class subset is entirelyrandom.

The balanced datasets, i.e. the majority data subset and the minoritydataset, are thereafter provided to an oversampling module 716, where aprocess of synthetic over-sampling with replacement (SMOTE) is carriedout on the first majority data subset and the minority data set tocreate a synthetic data set. In carrying out SMOTE, the over-samplingmodule 716 takes a data point in the first minority data set andintroduces synthetic examples along a line segment joining the datapoint to a predetermined number of data point neighbors. A syntheticdata set is formed with reference to both the majority data subset andthe minority dataset.

A training module 718 is provided, where data received by the HSS 260 istrained into classifiers such that incoming patient data may be receivedand evaluated against the trained classifier to provide a portion of arisk assessment score as to the health of a patient. In training module718, the balanced dataset is used to build a classification model, wherethe classification model, or classifier acts as an expert in providingan insight on whether the incoming patient is likely to have a positiveoutcome in relation to ACS.

After the classification models have been built, one classifier for eachparameter under analysis, the classification models are provided to avalidation module 720. Validation module 720 carries out a validation ofeach classification model by running its corresponding synthetic datasetthrough the classification model so as to obtain a resultant predictionaccuracy of the classification model, which represents the importance ofthe classifier. Each classification model is thus provided with itsprediction accuracy, after the validation module 720 has validated theclassification model with its corresponding synthetic dataset.

In the present embodiment, after the classification models are validatedby the validation module 720, the models are provided to a weighingmodule 722. Weighing module 722 carries out a sorting function on theclassification models based on its validation accuracy and arranges themin order, from highest to lowest, i.e. most important to leastimportant. The weighing module thereafter provides a weightage value toeach sorted classification model, set as its corresponding predictionaccuracy. In the present embodiment, after the classification sorting byvalidation accuracy, a rebuild of the classification model is carriedout with its corresponding dataset. In another embodiment, thepreviously built classification model for which the prediction accuracywas obtained for is reused.

An output of the classifier generation module 704 is thus that oftrained, validated and weighed classifiers 724, related to the variousparameters indicated as being of importance toward the assessment of arisk score by the triage system 100.

The HSS 260 is now ready for usage and assessment of an incomingpatient, and for providing a risk score therefrom. Analysis module 706includes a testing module 726 arranged to receive incoming patient testdata 728 from the central processor 104 of the present triage system100. In this embodiment, each received parameter of the incoming patienttest data 728 is tested with a corresponding trained classifier 724 toprovide a prediction output of whether the patient is likely to achievea negative or positive outcome. The prediction output is provided in abinary output format of 0 or 1.

After testing and evaluation of all the parameters and classifiers, theresultant prediction outputs are collated and passed on to a scoringmodule 730. Scoring module 730 calculates the risk score for theincoming patient, based on his incoming measurement parameters 728, as anormalized summation of the binary prediction outputs of all theweighted classifiers. This generated risk score provides the medicalpersonnel using the triage system 100 with a calibrated insight as towhether the incoming patient is at a risk of ACS.

Clinical Validation

A clinical study was carried out by the present applicants as to thevalidity of the presently disclosed triage system for use in emergencydepartments of hospitals in generating a risk score in assessingincoming patients for acute coronary syndromes. An observational cohortstudy of 564 critically ill patients with undifferentiated non-traumaticchest pain was conducted from March 2010 to March 2012. Patients werecomprised of a convenience sample presenting to the ED of SingaporeGeneral Hospital, the main acute tertiary hospital in Singapore, serving135,000 patients annually. Ethics approval with a waiver of patientconsent was obtained from the Institutional Review Board. In this study,recruited patients were adult men and women at least 30 years of age whopresented to the ED with a primary complaint of non-traumatic chestpain. Patients in non-sinus rhythm (e.g. asystole, supraventricular andventricular arrhythmias, complete heart block) and patients who weredischarged against medical advice or transferred to another hospitalwithin 72 hours of arrival at the ED were excluded. An eligible patientwho arrived at the ED with chest pain was randomly screened andrecruited by trained medical personnel.

The primary outcome was a composite of severe complications within 72hours of arrival at the ED, extracted from the electronic hospitalrecords. Patients were considered to have met the outcome if they had atleast one of the following severe complications: all-cause mortality,cardiac arrest, sustained ventricular tachycardia (VT), hypotensionrequiring inotropes or intra-aortic balloon pump (IABP) insertion,intubation or mechanical ventilation, complete heart block, andbradycardia requiring insertion of a pacing wire. In the compileddatabase, 19 out of 564 patients met the primary outcome.

Evaluation of the scoring system is based on the leave-one-outcross-validation (LOOCV) framework. In a dataset of K samples, Kiterations are required for algorithm evaluation. Within iteration, onesample is used as the testing sample while the rest samples are used fortraining. The proposed score prediction process needs to repeat K timesso that each sample can be tested individually. Having the risk scoresfor the entire dataset, a proper threshold is derived to reportsensitivity and specificity.

TABLE 1 Confusion matrix used for defining TP, TN, FP and FN Predictedacute cardiac Predicted complications within 72 h health Actual acutecardiac TP FN complications within 72 h Actual health FP TN

Table 1 shows a confusion matrix table used for defining true positive(TP), false positive (FP), true negative (TN), and false negative (FN).TP indicates patients with acute cardiac complications within 72 hcorrectly predicted as acute cardiac complications within 72 h; FPindicates healthy patients incorrectly predicted as cardiac arrestwithin 72 h; TN indicates healthy patients correctly predicted ashealthy; and FN indicates patients with acute cardiac complicationswithin 72 h incorrectly predicted as healthy. Thereafter, calculationsfor the sensitivity and the specificity of the system are provided asfollows:

$\begin{matrix}{{Sensitivity} = \frac{TP}{{TP} + {FN}}} & (5) \\{{Specificity} = \frac{TN}{{TN} + {FP}}} & (6)\end{matrix}$

Further, the receiver operation characteristic (ROC) curve, the positivepredictive value (PPV) and the negative predictive value (NPV) are alsoused to present system performance, where PPV and NPV are defined as:

$\begin{matrix}{{PPV} = \frac{TP}{{TP} + {FP}}} & (7) \\{{NPV} = \frac{TN}{{TN} + {FN}}} & (8)\end{matrix}$

A prospective, non-randomized, observational study to assess the utilityof combining 12-lead ECG, HRV and vital signs as a predictor of acutecardiac complications within 72 hours was conducted based on a cohort of564 patients with chest pain attended at the Department of EmergencyMedicine, Singapore General Hospital. USS and HSS algorithms using12-lead ECG, HRV and vital signs achieved area under the ROC curve (AUC)of 0.799 and 0.813, respectively, which were superior to both TIMI score(AUC of 0.621) and MEWS score (AUC of 0.672). The comparison results inTable 2 and FIG. 6 have shown the effectiveness of the proposed scoringsystems in the prediction of acute cardiac complications. The cutoffscores were selected to keep both sensitivity and specificity as high aspossible. Note that the ranges of scores for USS and HSS are 0-100 andthe ranges of scores for TIMI and MEWS are 0-6. We observe that PPVvalues are small while NPV values are large, which is due to the fact ofimbalanced data where negative class is the majority class. It is worthnoting that both USS and HSS algorithms are able to filter out 99%patients without acute cardiac complications, and this capability isuseful to conducting triage in critically ill patients. Furthermore,both USS and HSS can pick up 78.9% patients who met the primaryoutcomes, and meanwhile maintain high specificities (>73%).

TABLE 2 Prediction results with different scoring methods where inputsare feature vectors consisting of 12-lead ECG, HRV and vital signsScoring Cutoff AUC Sensitivity Specificity PPV NPV method Score (95% CI)(95% CI) (95% CI) (95% CI) (95% CI) USS 36.7 0.799 78.9% 73.6% 9.4%99.0% (0.677-0.920) (60.6%-97.3%) (69.9%-77.3%) (4.9%-14.0%)(98.0%-100.0%) HSS 50.6 0.813 78.9% 74.1% 9.6% 99.0% (0.694-0.931)(60.6%-97.3%) (70.5%-77.8%) (5.0%-14.2%) (98.1%-100.0%) TIMI 1.0 0.62178.9% 36.7% 4.2% 98.0% (0.484-0.757) (60.6%-97.3%) (32.7%-40.7%)(2.1%-6.2%)  (96.1%-99.9%)  MEWS 1.0 0.672 42.1% 78.5% 6.4% 97.5%(0.537-0.808) (19.9%-64.3%) (75.1%-82.0%) (2.9%-10.7%) (96.0%-99.0%) 

In the ROC analysis, we compared USS and HSS to both TIMI score and MEWSscore separately and recorded the comparison results in FIG. 8A and FIG.8B, respectively. FIG. 8A charts the performance of USS vs. TIMI andMEWS. FIG. 8B charts the performance of HSS vs. TIMI and MEWS. Ingeneral, both proposed USS and HSS outperformed TIMI and MEWS in termsof achieving higher AUC, sensitivity, specificity, PPV and NPV values.

In the present disclosure, an intelligent triage system with a novelscoring system to integrate 12-lead ECG, HRV and vital signs for riskprediction is described. An investigation was thus also carried out todetermine how the 12-lead ECG contributes to the system. Evaluationswere conducted on features with and without 12-lead ECG parameters withUSS and HSS algorithms and the comparison results are presented in Table3 and FIG. 9A and FIG. 9B. FIG. 9A charts the performance of USS withand without 12-lead ECG parameters. FIG. 9B charts the performance ofHSS with and without 12-lead ECG parameters. Compared to the scoringmethods with 12-lead ECG parameters, it is observed that methods without12-lead ECG parameters achieved lower AUC, specificity, PPV, and NPVvalues. In other words, 12-lead ECG may be a significant predictor ofacute cardiac complications.

TABLE 3 Results of different scoring methods with and without 12-leadECG parameters Scoring Cutoff AUC Sensitivity Specificity PPV NPV methodScore (95% CI) (95% CI) (95% CI) (95% CI) (95% CI) USS 36.7 0.799 78.9%73.6% 9.4% 99.0% (w/ ECG) (0.677-0.920) (60.6%-97.3%) (69.9%-77.3%)(4.9%-14.0%) (98.0%-100.0%) USS 20.0 0.729 78.9% 52.3% 5.5% 98.6% (w/oECG) (0.598-0.860) (60.6%-97.3%) (48.1%-56.5%) (2.8%-8.1%) (97.3%-100.0%) HSS 50.6 0.813 78.9% 74.1% 9.6% 99.0% (w/ ECG)(0.694-0.931) (60.6%-97.3%) (70.5%-77.8%) (5.0%-14.2%) (98.1%-100.0%)HSS 33.4 0.730 78.9% 53.8% 5.6% 98.7% (w/o ECG) (0.599-0.861)(60.6%-97.3%) (49.6%-57.9%) (2.9%-8.4%)  (97.3%-100.0%)

In addition, upon reviewing further the results reported in Table 2 andTable 3, it was found that both USS and HSS still outperformed TIMI andMEWS scores even though 12-lead ECG was not used for prediction. Thisobservation provided more evidence on the effectiveness of the proposedUSS and HSS algorithms in predicting acute cardiac complications.

Although the following description of particular system embodiments isdirected at a triage system 100 for determining a risk score in ahospital emergency department, it will be understood that the system 100according to various embodiments of the present disclosure canadditionally, or alternatively, be used for carrying out a determinationof a risk score of a patient under any other circumstance, within thescope of the present disclosure.

For example, an embodiment is envisioned wherein a cardiac eventassessment system is provided for home or other (e.g. office) use. Thehome-use cardiac event assessment system would include a generallycentral ensemble-based scoring system, which is envisioned as a portableassessment system, configured for ease of use without a trained medicalpractitioner. Such a system would be limited such that a cardiac eventrisk score is provided as the assessment output, the risk scoreproviding an insight as to severity of any present chest pain, and forencouraging home users to take the relevant medical action.

Such a system is envisioned to include data connectivity to the internetand thereafter establish a secure data connection with the hospital pastpatient data database. Classifiers in the scoring system of the cardiacevent assessment system are pre-built for the home-user, such that inthe event of use, a result can be provided more expediently.

In home use, vital signs such as heart rate, respiratory rate, bloodpressure, or SpO₂ readings can be taken and provided as input to thecardiac event assessment system. Further, to provide an even moreaccurate present diagnosis an ECG can be taken, where the results can beanalyzed with an ECG analysis module and an ECG parameter and/or a HRVparameter provided to the cardiac event assessment system. Analysis canthus be carried out quickly against the pre-built classifiers and a riskscore as to a cardiac event can be provided.

In home use, another embodiment is envisioned wherein the cardiac eventassessment system is provided for home use. The home-use cardiac eventassessment system includes an emergency alarm system. In the event of anemergency where the risk score goes beyond a pre-built acceptable riskscore provided to the cardiac event assessment system, the emergencyalarm system will trigger an alarm and with internet connectivity or 3Gor 4G or Wifi, the alarm will transfer the risk score to the trainedmedical practitioner who is holding a portable handheld device capableof receiving the risk score. The trained medical practitioner can thenimmediately return to the hospital and provide evasive rescue to thepatient.

Aspects of particular embodiments of the present disclosure address atleast one aspect, problem, limitation, and/or disadvantage associatedwith existing hospital ED triage systems. While features, aspects,and/or advantages associated with certain embodiments have beendescribed in the disclosure, other embodiments may also exhibit suchfeatures, aspects, and/or advantages, and not all embodiments neednecessarily exhibit such features, aspects, and/or advantages to fallwithin the scope of the disclosure. It will be appreciated by a personof ordinary skill in the art that several of the above-disclosedsystems, components, processes, or alternatives thereof, may bedesirably combined into other different systems, components, processes,and/or applications. In addition, various modifications, alterations,and/or improvements may be made to various embodiments that aredisclosed by a person of ordinary skill in the art within the scope andspirit of the present disclosure. Such different systems, components,processes and/or modifications, alterations, and/or improvements areencompassed by the following claims.

1. A system for determining a risk score for triage, comprising: a firstinput device for measuring a first input parameter relating tophysiological data of a patient, the first input parameter comprising avital signs parameter; a twelve-lead electrode electrocardiogram (ECG)device, for carrying out a electrocardiography procedure on the patient,and establishing an ECG obtained from results of the electrocardiographyprocedure, the ECG device comprising an ECG extraction module to extractat least one ECG parameter from the ECG; a heart rate variability (HRV)analysis module for determining a HRV analysis from the ECG, the HRVanalysis comprising at least one HRV parameter; and an ensemble-basedscoring system, comprising: a plurality of weighted classifiers forproviding a risk score calculation, the plurality of weightedclassifiers established based on past patient data in a database ofaccumulated past patient data; and an analysis module for receiving thefirst input parameter, the at least one HRV parameter, and the at leastone ECG parameter which are communicated or transmitted to theensemble-based scoring system, wherein the analysis module determines arisk score by comparing the first input parameter, the at least one HRVparameter, and the at least one ECG parameter to corresponding weightedclassifiers.
 2. A system for determining a risk score as claimed inclaim 1, wherein the ensemble-based scoring system further comprises adata access module for obtaining past patient data and configured fordata communication with the database of accumulated past patient data.3. A system for determining a risk score as claimed in claim 1, whereinthe ensemble-based scoring system further comprises a sorting modulearranged to: receive data from the database of accumulated past patientdata; and sort the data into a plurality of data sets, each data setcorresponding to a classifier, and comprising an imbalanced data set. 4.A system for determining a risk score as claimed in claim 3, wherein theensemble-based scoring system further comprises a sampling modulearranged to: receive a first imbalanced data set corresponding to afirst classifier comprising a first majority data set comprising a firstnumber of data samples, and a first minority data set comprising asecond number of data samples, from the sorting module; and extract afirst majority data subset comprising a third number of samples from thefirst majority data set; wherein the third number of samples in thefirst majority data subset is equal to the second number of samples inthe first minority data set.
 5. A system for determining a risk score asclaimed in claim 4, wherein the ensemble-based scoring system furthercomprises a classifier generation module for establishing the pluralityof weighted classifiers, based on past patient data provided by the dataaccess module.
 6. A system for determining a risk score as claimed inclaim 5, wherein the classifier generation module further comprises atraining module arranged to: receive the first majority data subset andthe first minority data set from the sampling module; and build a firstclassification model to represent the first classifier with the firstmajority data subset and the first minority data set.
 7. A system fordetermining a risk score as claimed in claim 6, wherein the trainingmodule receives a plurality of majority data subsets and minority datasets from the sampling module; and builds classification modelsrepresenting a plurality of classifiers with the received plurality ofmajority data subsets and minority data sets.
 8. A system fordetermining a risk score as claimed in claim 4, wherein theensemble-based system further comprises an over-sampling module arrangedto: receive the first majority data subset and the first minority dataset from the sampling module; and create a first synthetic data set byapplying a process of synthetic over-sampling with replacement on thefirst majority data subset and the first minority data set.
 9. A systemfor determining a risk score as claimed in claim 8, wherein theensemble-based system further comprises a validation module arranged to:build a first classification model with a training module based on thefirst majority data subset and the first minority data set correspondingto the first classifier; validate the first classification model againstthe first synthetic data set; and obtain a resultant prediction accuracyof the first classification model, representing the importance of thefirst classifier.
 10. A system for determining a risk score as claimedin claim 9, wherein the over-sampling module receives a plurality ofmajority data subsets and minority data sets from the sampling moduleand creates a plurality of synthetic data sets; the training modulebuilds a plurality of classification models representing a plurality ofclassifiers with the received plurality of majority data subsets andminority data sets; the validation module validates the plurality ofclassification models against the plurality of synthetic data sets, andobtains a plurality of prediction accuracies of the classificationmodels, representing the importance of each of the plurality ofclassifiers.
 11. A system for determining a risk score as claimed inclaim 10, wherein the classifier generation module further comprises aweighing module for allocating each of the plurality of classifiers witha weightage according to its importance, to obtain the plurality ofweighted classifiers.
 12. A method of determining a risk score,comprising: measuring a first input parameter relating to physiologicaldata of a patient, the first input parameter comprising a vital signsparameter; carrying out a twelve-lead electrocardiography procedure onthe patient; establishing a ECG from results of the electrocardiographyprocedure, the ECG comprising at least one ECG parameter extractablefrom the ECG; extracting the at least one ECG parameter from the ECG;determining a heart rate variability (HRV) analysis from the ECG, theHRV analysis comprising at least one HRV parameter; providing the firstinput parameter, the at least one HRV parameter, and the at least oneECG parameter to an ensemble-based scoring system; the ensemble-basedscoring system comprising a plurality of weighted classifiers forproviding a risk score calculation, the plurality of weightedclassifiers established based on past patient data in a database ofaccumulated past patient data; and determining a risk score with theensemble-based scoring system by comparing the first input parameter,the at least one HRV parameter, and the at least one ECG parameter tocorresponding weighted classifiers.
 13. A method of determining a riskscore as claimed in claim 12, wherein the at least one ECG parameter isany one of a ST elevation, a T wave inversion, a Q wave, a QT intervalcorrection (QTc), a QRS axis, a left bundle branch block (BBB), a rightBBB, an IntraVentricular Conduction Delay (IVCD), a left atrialabnormality (LAA), a left ventricular hypertrophy (VH), a right VH, andan atrial fibrillation.
 14. A method of determining a risk score asclaimed in claim 12, comprising extracting a plurality of RR intervalsfrom the ECG and performing any one of a time domain analysis and afrequency domain analysis to obtain the at least one HRV parameter. 15.A method of determining a risk score as claimed in claim 14, wherein theat least one HRV parameter is any one of an average length of the RRintervals, standard deviation of all RR intervals, a mean heart rate, astandard deviation of all instantaneous heart rate values, a NN50 count,a pNN50 percentage, a square root of mean squared differences ofsuccessive RR intervals, a HRV triangular index, a baseline width oftriangular fit into a RR interval histogram, a total power, a very lowfrequency power, a low frequency power (LF), a high frequency power(HF), a normalized low frequency power, a normalized high frequencypower, and a ratio of LF/HF.
 16. A method of determining a risk score asclaimed in claim 12, wherein the first input parameter is any one of aheart rate, a respiratory rate, a blood pressure reading, a temperaturereading, a Glasgow Coma Score (GCS), an oxygen saturation reading, and apain score.
 17. A method of determining a risk score as claimed in claim12, further comprising establishing a second input parameter relating toa medical status of the patient, providing the second input parameter tothe ensemble-based scoring system, and determining the risk score byfurther comparing the second input parameter to a corresponding weightedclassifier.
 18. A method of determining a risk score as claimed in claim17, wherein the second input parameter is any one of a medical history,a drug history, a smoking history, a family history of heart disease,and a number of angina events in the past 24 hours.
 19. A method ofdetermining a risk score, comprising: receiving a first imbalanceddataset corresponding to a first classifier and sampling the datasamples to form a first balanced data set; creating a first syntheticdata set by applying a process of synthetic over-sampling withreplacement on the first balanced data set; building a firstclassification model based on the first balanced data set correspondingto the first classifier; validating the first classification modelagainst the first synthetic data set; and obtaining a resultantprediction accuracy of the first classification model, representing theimportance of the first classifier.
 20. A method of determining a riskscore as claimed in claim 19, further comprising: obtaining a pluralityof prediction accuracies relating to a plurality of classificationmodels; sorting the plurality of classification models according to itsprediction accuracy; allocating each of the plurality of classificationmodels with a weightage according to its importance to obtain aplurality of weighted classifiers; evaluating an input parameter withits corresponding weighted classifier; and generating a binaryprediction output of either 0 or 1 for each evaluated weightedclassifier; and calculating the risk score based on a normalizedsummation of the binary prediction outputs of all evaluated weightedclassifiers.